Design Network Telemetry & Monitoring

Understanding the problem

This is the design that reads as most recognizably Cisco, and it helps to see why before touching a single box. Cisco's business is the network itself, the routers and switches and access points that carry the traffic, plus the software that watches over them. A platform that pulls health and counters off a huge fleet, then keeps that history and pages someone when a link drops, is not a hypothetical here. It is ThousandEyes. It is the Meraki dashboard. It is the observability layer inside their own products. The interviewer is not checking whether you can recite a generic ingestion diagram. They want to know whether you understand the shape of a system they ship.

How much the design round weighs depends on the level you are chasing. For sub-senior candidates it is a light touch, a few concept checks folded into a technical screen. From senior up it turns into a genuine, level-gating conversation. It usually opens on a system you have actually shipped, then pivots to a prompt like this one, with the interviewer acting as a fellow architect rather than an examiner. The prompt starts manageable and grows under you. Now it is not a hundred devices but two hundred thousand. Now a core link just failed and everything behind it went dark. Now the store is falling over. Expect the push to land on the parts a networking company feels in its bones. The firehose of writes. The cardinality of the data. And what a monitoring system does during the exact outage it exists to catch.

The Cisco angle

At Cisco, a network monitoring platform describes their own product line, so the bar sits higher than a textbook pipeline. ThousandEyes watches network paths across the internet. Meraki streams device health into a cloud dashboard. Model-driven telemetry ships inside IOS-XE and NX-OS. An interviewer here has almost certainly built or operated a piece of this, which means a generic answer gets probed fast and a domain-aware one lands hard. Treat the network as the source of the hard parts. The fleet is huge. Faults spread through a topology in correlated waves. And the monitoring itself adds load to devices that are supposed to be busy forwarding packets. Design for that world and every box you draw has a reason a networking engineer will recognize.

Functional requirements

Keep the core small and let the interviewer stretch it. There are four jobs. First, ingest metrics from a large fleet on a short interval: interface counters, CPU and memory, temperature, protocol and link health. Second, store that history as time-series and keep it queryable over a long retention window, with older data downsampled to stay affordable. Third, let an operator query any metric and chart it, sliced however the data is labeled, by device or by interface or across a whole region. Fourth, evaluate alert rules continuously and notify the right people fast. One requirement carries more weight than it looks. The platform has to accept both pull and push. A real fleet is always a mix: some devices stream their own telemetry, and others you still have to poll.

Non-functional requirements

Write throughput leads, because the defining fact of this system is a relentless append of new samples from a vast fleet, and everything downstream has to keep up with it. Retention comes next, and it is really a cost target: keep a long history without keeping every sample at full resolution forever, which means downsampling old data. Then availability, framed as graceful degradation, because a fleet this size always has something broken. A collector process will die, a device group will drop off, and the platform has to keep ingesting the healthy majority while that heals. Last is alert latency. A link goes down and someone should know in seconds, not on the next slow polling cycle, so the alerting path has to run on a much tighter clock than storage or query.

Bank this slack now: a single sample is cheap and slightly lossy is fine. Miss one reading of an interface counter on a ten-second stream and the next one arrives ten seconds later, carrying the same running total, so nothing is truly lost. That means under pressure the platform can drop a reading, sample fewer of them, or downsample the history, all without breaking its promise. Per-sample exactness is not the point. Keeping up with the firehose and paging fast are. We will spend that slack deliberately later, on downsampling old data and on letting a queue shed a burst rather than stalling ingestion.

• ~200K devices monitored. Routers and switches, plus firewalls and access points, across an enterprise or service-provider estate. Illustrative teaching figure, not an audited count.
• ~10M/sec samples ingested at peak. A hundred million series on a ten-second streaming interval. This is the write firehose the whole design has to absorb.
• ~100M active time-series. Device times interface times counter. The cardinality figure, and the number that decides whether the store survives.
• seconds alert latency target. A link fails and the page should land in seconds, which is why alerting runs on the stream and not on a scheduled query.

Sizing the firehose

Put the firehose on the board before the architecture, because its size dictates almost every later choice. Treat every figure as an illustrative teaching estimate and derive it out loud, since a good interviewer redoes the arithmetic with you. Start with two hundred thousand devices. A mid-size access switch has forty-eight ports, and each port carries a handful of counters worth watching: bytes in and out, packets, errors, discards, utilization, call it ten per interface. Forty-eight ports is roughly four hundred and eighty interface counters. Another twenty or so device-level metrics (CPU, memory, temperature, a few protocol neighbor states) round it to about five hundred series per device. Multiply. Two hundred thousand devices at five hundred series each is one hundred million distinct time-series. Hold onto that number. It comes back to bite twice, once in storage and once in alerting.

Now turn series into a write rate, which is exactly where pull and push part ways. Poll every series once every thirty seconds, the old SNMP cadence, and you are writing a hundred million samples every thirty seconds, about three and a third million a second. Push instead on a ten-second streaming interval and it is ten million a second. Ask for one-second granularity on everything and it is a hundred million a second, which is the figure that explains why nobody streams every counter at one second. The interval is a dial that swings the write rate by an order of magnitude. Choosing it per metric, fast for the counters that matter and slow for the ones that rarely move, is one of the first levers you reach for.

• The write firehose: ~10M samples/sec. a hundred million series on a ten-second interval. It is append-heavy and timestamped and never stops, which is why a relational database is the wrong tool and a queue sits in front of storage.
• Cardinality: ~100M active series. device times interface times counter. Add one careless label (per-flow, per-neighbor) and it multiplies again. This is the number that quietly kills a naive store.
• Storage growth: ~1.7 TB/day compressed. ten million samples a second even at a couple of bytes each. Keeping every sample at full resolution forever does not scale, so old data is rolled up.
• Reads vs writes: writes steady, reads spike. ingestion is a constant firehose; a dashboard load or an incident fires a burst of heavy range queries. A big query must never stall ingestion.

Two lines in that grid set the agenda. The firehose has to be ingested and stored cheaply, and the two traps hiding inside it, cardinality and the alert storm, are where a networking system either holds or breaks. So the design keeps a few things true at once. The write path stays cheap and mostly hands-off. The store stays bounded as the fleet grows. And a single correlated failure never turns a steady stream of data into a flood of pages.

The set-up

Core entities

Name the objects before drawing anything. Four carry this design, and the one to pin down first is the series, because its count is the cardinality that governs the whole store.

Pin the stores next, because the interviewer will ask where each piece lives. Samples land in a time-series database, keyed by series and ordered by time, because that is the one access pattern that matters: append the newest point, read a range for one series. The alert rules and their firing state live in a small durable store the alert engine reads, since there are only thousands of rules and they change rarely. A series index maps a human query ("every interface on devices in Frankfurt") to the underlying series ids, so a dashboard resolves a filter without scanning everything. The message queue in the middle, Kafka, is not a store at all. It is a buffer that decouples the collectors from the database, so a slow write never pushes back onto a device.

API

Keep the external surface thin, and let the shape of each call say which regime it belongs to. Ingestion is not a request per sample; it is a long-lived stream from a device or a poll the collector drives. The query side is a range read by time. Reference these by name in the deep dives.

Two of these carry the design. The streaming ingest is the firehose. Everything about it exists to move millions of samples a second without the platform asking for each one: the device pushes and the collector simply receives, so the cost per sample collapses. The query call is the mirror image: a range read by time that a dashboard fires in spiky bursts, and a broad one can starve writes if the two are not kept apart. The alert notification is the payoff of the whole system. Its note hides the hardest problem in the design: what "grouped and deduplicated" has to mean when a single failure trips ten thousand rules at once.

High-level design

Follow the data one hop at a time. Metrics leave the Network devices two ways, pushed on a stream or pulled by a poll, and reach the Collector fleet, which normalizes them and publishes onto the Telemetry queue. From that queue two consumers read the same records: the Time-series store writes them down, and the Alert engine evaluates them against its rules in near real time. Operators reach the history through Query + dashboards, which range-reads the store, and a firing alert leaves through On-call + webhooks. The diagram colors each box by role: blue for the devices and the on-call destination, violet for services, amber for the queue, green for the store. The walkthrough reuses the same tags.

1) Devices report, two ways. The Network devices hand over metrics either by streaming them (the device pushes on a subscription over gRPC dial-out) or by being polled (the collector drives an SNMP round-trip on a timer). Both land at the Collector fleet. Keeping both paths is deliberate, because the fleet is a mix and the platform cannot pretend every device can stream. The bottleneck being set up here, and defused in the first deep dive, is that this ingest path carries the entire firehose.

2) Collectors normalize and buffer. Each collector in the Collector fleet owns a shard of the fleet, turns whatever it receives into a common sample shape, and publishes onto the Telemetry queue. The queue matters more than it looks. It decouples ingest from storage, so a burst or a slow database queues durably in the log instead of pushing back onto the devices or dropping on the floor. What is punted here is how the collectors divide the fleet and survive one of their own dying, which the last deep dive owns.

3) The store writes the firehose. The Time-series store consumes the Telemetry queue and appends each sample to its series, ordered by time. This is where retention and cost get decided. Recent data stays at full resolution on fast disk, old data is rolled up into coarser aggregates, and the whole thing is partitioned so it scales by adding nodes. The trap being deferred is cardinality, the number of distinct series, which is the second deep dive.

4) The alert engine evaluates on the stream. In parallel with the store, the Alert engine reads the same Telemetry queue and checks samples against its rules as they arrive. A breach is seen seconds after it happens rather than on the next scheduled query. It never touches the storage database, which keeps alerting and ingestion from fighting over the same resource. The hard part it defers is what to do when one failure trips ten thousand rules in the same instant, which the third deep dive is all about.

5) Operators read the history. Query + dashboards resolves a filter through the series index and range-reads the Time-series store for a chart or an investigation. This path is spiky, quiet until an operator opens a dashboard or an incident lights up a room, and then it fires a burst of heavy queries at the worst moment. Keeping that burst from stalling the write firehose, by isolating the read tier, is a thread the last deep dive picks up.

Technology choices

A labeled box proves nothing on its own. Back it with a real tool, argue why that kind of tool suits this workload, and know where it buckles under this load. Slot a mainstream choice into each one, and the follow-ups will all come from the failure modes.

• Telemetry queue is a durable log: Kafka, partitioned so the load spreads across brokers and each partition keeps its records in order. A log fits because ingest is bursty and has two independent consumers at different speeds. The store and the alert engine each track their own offset and replay after a crash. The failure story is a consumer falling behind. It doesn't lose data; it reads older offsets and catches up. Telemetry tolerates a little loss, so if a consumer is badly behind, the oldest samples can age out of retention without breaking anything.
• Time-series store is a purpose-built time-series database: Prometheus, InfluxDB, or TimescaleDB, chosen because the access pattern is append the newest point and read one series by range. This category compresses timestamps and slowly-changing values into a couple of bytes each and serves range-by-time reads cheaply, which is exactly the workload. The failure mode to name is cardinality: memory and index scale with the number of active series, so a hundred million series is a design constraint and a careless label is an outage.
• The ingest transport is gRPC dial-out streaming telemetry for the push path and SNMP for the pull path. Streaming is right at this scale because the device sends on its own, so the platform stops paying a round-trip per sample and the interval can drop toward seconds. SNMP stays because a large slice of any real fleet only speaks it. The failure to plan for is a reconnect storm when a group of devices comes back at once and all re-establish their streams, which the collectors and the queue have to absorb.
• Alert engine is a streaming rule evaluator. It's a stateful stream processor (a Flink or Kafka-Streams style consumer, or the streaming-alerting core of a monitoring stack) that holds each rule's state in memory and evaluates against the sample flow. Streaming is right because the alert latency budget is seconds and a scheduled query cannot meet it. The failure mode is the alert storm, and the answer is the grouping and inhibition layer, which is where this component earns its place.

The Cisco angle

Map the design onto Cisco's own stack when the interviewer asks, and be specific about the push side. Model-driven telemetry, a device streaming structured data defined by a YANG model over gRPC dial-out, ships in IOS-XE and NX-OS. It exists because SNMP polling couldn't keep up with the fleets and intervals operators wanted. None of these products hands you the whole platform. The device hands over a clean feed. But the collector sharding and the cardinality budget, the retention tiers and the alert grouping, all of that is still yours to design. Name model-driven telemetry as the modern ingest path, and keep SNMP as a path you still support rather than sneer at. That shows you know their world without pretending the platform does the hard part.

Potential deep dives

Everything still on the table is won here, in the dives. A telemetry round almost always pushes on the same four corners. How do the metrics get in, where do the time-series live, how does alerting stay sane, and what happens at full scale when a piece fails? The two traps a networking company reaches for fastest are cardinality and the alert storm, so those get the most room.

1) Ingestion at scale: from polling to streaming telemetry

Start at the wire, because how a device hands over its metrics is the choice the modern version of this system turns on. For decades the answer was polling: a monitoring server walks a list of devices and asks each one for its counters on a timer, usually over SNMP. It works, and plenty of the fleet still speaks only that language. But polling hundreds of thousands of devices every few seconds runs into a wall, and the industry has been moving the other way, toward letting the device push its own telemetry. Walk the arc, because the interviewer wants to see you arrive at push for the right reasons rather than just name it.

The Cisco angle

To a Cisco interviewer, the move from polling to streaming telemetry is a shift their own platforms drove, not a generic observability fashion. Model-driven telemetry, a device streaming structured data over gRPC dial-out, ships in IOS-XE and NX-OS. It exists because SNMP polling couldn't keep pace with the fleets and intervals operators wanted. If you reach for push and can say why, the round-trip cost, the control-plane load on the device, the interval floor, you are describing the exact evolution their engineers lived through. Naming SNMP as the pull path you still support, rather than something to be embarrassed by, also reads as real. A live fleet is always a mix, and the platform has to speak both languages without apology.

2) Time-series storage and the cardinality trap

Now the firehose has to land somewhere, and the instinct to reach for a familiar relational database is the first thing to talk yourself out of. The access pattern here is narrow and peculiar. Writes are almost all appends of the newest point, reads are almost all a range of one series over a time window, and there are a hundred million series. A store built for that looks nothing like a store built for transactional rows. Walk why, because the reason a time-series database exists is the reason your first instinct is wrong.

3) Alerting: stream evaluation and the alert storm

Storage lets you look at the past; alerting is what pages someone about the present, and it runs on a tighter clock. An alert that fires two minutes after a core link drops is nearly useless. Evaluation has to happen on the stream of samples as they arrive, ahead of any query that trails behind. There are three shapes of rule worth supporting, and one failure mode that dominates the entire dive.

A threshold is the simplest rule: fire when a value crosses a line, CPU over ninety percent, an interface past eighty percent utilization, temperature over a limit. A rate-of-change rule catches trouble a static line misses, like input errors on an interface climbing fast even while the absolute count is still low, because the slope is the early warning. An anomaly rule compares a series against its own recent baseline and fires when it drifts far outside its usual band. It catches the problems nobody thought to write a threshold for. All three are cheap to evaluate on one series. The hard part is what happens when thousands of series trip in the same instant.

The Cisco angle

The alert storm is the part of this design a networking company cares about most, because it is the failure they live through for real. In a network, faults correlate by topology. One core link or an upstream router failing takes ten thousand devices unreachable in the same second. A monitoring system that pages ten thousand times has actively made the outage harder to fix. The senior answer knows the naive stream detector is correct and useless here, and reaches for grouping and inhibition, one incident for the core link and the rest suppressed beneath it. Anyone who has carried a pager for a network has felt this. The goal is one actionable page, not a true-but-useless flood. Saying that plainly reads as someone who has run the thing rather than only drawn it.

4) Scale and failure: the firehose, a dead collector, query isolation

The last dive is the staff-level one: what the numbers actually are, and what breaks when a piece falls over. Re-derive the firehose so it is on the board, because every failure conversation references it. A hundred million series on a ten-second interval is ten million samples a second, steady, all day and all night. That single figure explains three design facts. A queue sits in front of storage, the store is a horizontal cluster rather than one box, and a broad query is dangerous. No single machine is the answer to anything here, so the design is a set of independently scaled tiers with a buffer between them.

Take query isolation first, because it is the one people forget. Ingestion is a relentless, predictable firehose. Queries are the opposite: mostly quiet, then spiky. An operator opening a dashboard or an incident lighting up a war room fires a burst of heavy range queries at once, exactly when ingestion is also peaking. If reads and writes share the same nodes, one expensive query can starve the ingest path. A year of data across every interface in a region is enough to do it. The platform then falls behind on writes during the incident you most need it standing. So you isolate them. Route queries to read replicas or a separate query tier that reads the same storage, cap and time out expensive queries, and keep the write path on nodes a dashboard cannot reach. Ingestion must never lose to a dashboard.

Now the collectors, because a dead collector is the most common failure and the interviewer will ask how the fleet divides work and heals. The question underneath is how much has to move when one collector joins or leaves. The answer separates a design that stampedes from one that barely notices.

• A collector dies: one shard re-homes. consistent hashing moves only that collector's devices to its neighbors; the coordinator drives the rebalance and the rest of the fleet never moves.
• The queue absorbs bursts: buffer, do not drop. a streaming burst or a writer slowdown queues durably in Kafka instead of pushing back onto devices, so ingestion smooths out rather than shedding.
• Query vs ingest: isolate the paths. reads go to a separate query tier or replicas, expensive queries are capped, and the write path stays on nodes a dashboard cannot reach.
• The store scales out: shard by series + time. no single node holds the firehose; series and time partitioning spread the writes, and old partitions compact or drop as whole units.

What is expected at each level

This design carries a lot of surface, and the rungs separate on how much of it you own before anyone asks. Below senior the round is a light touch. At senior and above it becomes a gating discussion, usually leading with a system you've operated before it turns to a prompt like this one. Read the rungs as a climb: a working pipeline at mid, the pull-to-push shift and the cardinality trap owned at senior, and the alert storm and the failure story carried at staff.

Mid-level engineer

Spend your time on breadth, about eighty to twenty, and reach one coherent end-to-end path. Devices report metrics to a collector fleet, the collectors publish to a queue, a time-series database stores the samples, and an alert engine and a dashboard read from there. Name the right store, a time-series database and not a relational one, and be able to say why the access pattern demands it. The phase that earns the pass is scale. When the interviewer asks what happens at hundreds of thousands of devices, reason toward sharding the collectors and putting a queue in front of storage rather than one server doing it all. A working pipeline you can defend, with the interviewer filling in the harder corners, clears this rung.

Senior engineer

Now the balance tips to depth, roughly sixty to forty, and this is the level the round is calibrated to. Raise the hard facts yourself. Derive the firehose out loud, devices times series over the interval, and use it to justify the queue and the horizontal store. Make the pull-versus-push call and explain why streaming telemetry beats polling at this scale, while keeping SNMP as the path for the devices that need it. And surface cardinality without being asked: the store scales with the number of series, and device times interface times counter is where a naive design dies. Reaching a near-complete design whose write path and store choice and cardinality risk you flagged on your own is the bar here.

Staff and beyond

At this level depth dominates the whole picture, about forty to sixty. Own the alert storm end to end: a correct stream detector produces ten thousand pages when a core link fails, and grouping plus inhibition is what collapses that into one actionable incident. Carry the failure story, a collector death re-homing one shard through consistent hashing, the queue absorbing bursts, query and ingest isolated so a dashboard never stalls the firehose. Hold retention tiers, rollups and cardinality discipline as deliberate levers, and know how an operator changes a rule, watches a device pinned at a threshold, and sees the store's own cardinality climbing. You drive the design, and the interviewer course-corrects instead of stepping in to save it. The bar is holding ingestion, storage, alerting and failure as four separate levers while keeping write throughput and near-real-time alerting intact through every refinement.